Image signal processing device

ABSTRACT

An image signal processing device generates first image data and second image data based on the same optical image. Each pixel of the second image data is offset from the corresponding pixel of the first image data by half the distance between the centers of two adjacent pixels. The first image data is then subjected to a discrete cosine transformation (DCT), quantization, and Huffman encoding, and is recorded to an IC memory card. High resolution image data is generated based on the first and second image data. Expanded image data is obtained based on the first image data, and supplementary data is generated based on the high resolution image data and the expanded image data. The supplementary data is subjected to DCT, quantization and Huffman encoding, and is recorded to the same IC memory card.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image signal processing device bywhich digital still image data of different resolutions are recorded ona recording medium, and by which the recorded still image data are readfrom the recording medium so that two images having the differentresolutions can be reproduced.

2. Description of Related Art

Conventionally, there is known an electronic still camera in which astill image signal is converted to an electrical signal by an imagesensor (CCD) and recorded on a magnetic disk or an optical discaccording to the NTSC system, or another system, such as Hi-Vision (ahigh definition television system). The resolution of the NTSC system islower than that of a high definition television system, and therefore,even if a first image signal recorded according to the NTSC system isconverted to a second image signal according to the high definitiontelevision system, the resolution of the first image signal is notimproved.

On the other hand, two kinds of image signals having differentresolutions from each other can be recorded on a recording medium.According to such a construction, however, the amount of image signalrecorded on the recording medium will be increased, and thus, it isdifficult to record many images on a recording medium which has a fixedrecording volume.

SUMMARY OF THE INVENTION

Accordingly, the present invention has an object to provide an imagesignal processing device which can record image signals having differentresolutions in a recording medium without greatly increasing the amountof image signal on the recording mediums and can output the imagesignals so that a display device shows the images at the correspondingresolutions.

According to the present inventions there is provided an image signalprocessing device comprising first generating means, second generatingmeans, third generating means, transforming means, subtracting means,data compression applying means, and recording means.

The first generating means generates first image data based on anoptical image. The first image data has a first resolution. The secondgenerating means generates second image data based on the optical image.The second image data is offset from the first image data by apredetermined amount on a spatial coordinate axis. The third generatingmeans generates third image data based on the first and second imagedata. The third image data has a second resolution higher than the firstresolution. The transforming means transforms the first image data intoexpanded image data having the same number of pixels as the third imagedata. The subtracting means subtracts the expanded image data from thethird image data to generate supplementary data. The data compressionapplying means applies data compression to the first image data and thesupplementary data. The recording means records the first image data andthe supplementary data compressed by the recording means on a recordingmedium.

Further, according to the present invention, there is provided an imagesignal processing device comprising an image sensor, an A/D converter, afirst expansion circuits a second expansion circuit, a subtractor, adata compression circuit, and a recording medium.

The image sensor generates an image signal corresponding to a subjectand outputs image data. The A/D converter A/D converts the image data togenerate first image data and second image data, each of which has afirst resolution, and which are offset to each other by a predeterminedamount on a spatial coordinate axis of the original image data. Thefirst expansion circuit generates third image data based on the firstand second image data. The third image data has a second resolutionhigher than the first resolution. The second expansion circuit performsan expansion process to transform the first image data to expanded imagedata that has the same number of pixels as the third image data. Thesubtractor subtracts the expanded image data from the third image datato generate supplementary data. The data compression circuit appliesdata compression to the first image data and the supplementary data. Thefirst image data and the supplementary data compressed by the recordingmeans are recorded in the recording medium.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be better understood from the description ofthe preferred embodiments of the invention set forth below, togetherwith the accompanying drawings, in which:

FIG. 1 is a block diagram of a recording system of an electronic stillcamera to which a first embodiment of the present invention is applied;

FIG. 2 is a block diagram of a reproducing system of the electronicstill camera having the recording system shown in FIG. 1;

FIG. 3 is a view showing an example of DCT (discrete cosinetransformation) and quantization;

FIG. 4 is a view showing a group classification table of DC componentsused for Huffman encoding;

FIG. 5 is a view showing code words expressing group numbers;

FIG. 6 is a flowchart showing a processing routine of a Huffman encodingof AC component;

FIG. 7 is a view showing a zigzag scanning used when encoding ACcomponents among the DCT coefficients;

FIG. 8 is a view showing an example of finding the encoded data from thequantized DCT coefficient;

FIG. 9 is a view showing an example of decoding the image data by IDCT(inverse discrete cosine transformation) and inverse quantization fromthe encoded data;

FIG. 10 is a view showing an example of a high resolution image datagenerated based on first image data and second image data, andsupplementary data, according to an embodiment of the first invention;

FIG. 11 is a block diagram of a recording system of an electronic stillcamera to which a second embodiment of the present invention is applied;and

FIG. 12 is a view showing an example of data obtained by upsamplingrestored image data in a horizontal direction and a vertical directionaccording to the hierarchical process of the JPEG algorithm.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention will now be described with reference toembodiments shown in the drawings.

FIG. 1 is a block diagram of a recording system of an electronic stillcamera to which a first embodiment of the present invention is applied.

Light coming from an object S to be photographed is focused by a focuslens 11. The optical image is focused on a light receiving surface of aCCD (charge coupled device) or an image sensor 12. A half-mirror 13 isprovided between the focus lens 11 and the CCD 12, so that a part oflight focused by the focus lens 11 is reflected and focused on a lightreceiving surface of a CCD or an image sensor 14.

A large number of photoelectric conversion elements are arranged on thelight receiving surfaces of the CCDs 12 and 14. Each photoelectricconversion element corresponds to one pixel. The CCDs 12 and 14 arearranged in such a manner that a pixel of the electric conversionelement of the CCD 12 and a pixel of the CCD 14 are offset to each otherby half the distance between the centers of two adjacent pixels in ahorizontal direction of the image formed on the CCDs 12 and 14.

The optical image generated on the CCD 12 is converted to an electricalsignal by the photoelectric conversion elements and inputted to an A/Dconverter 15, so that the analog image signal is converted to firstdigital image data D10 for each pixel. Similarly, the analog imagesignal outputted from the CCD 14 is converted to second digital imagedata D14 for each pixel by an A/D converter 16. Each of the first andsecond image data has a first resolution according to a normaltelevision mode such as the NTSC system.

The first image data D10 is subjected to a predetermined processing by asignal processing circuit, (not shown) and then, one frame's worth (orone field's worth) of image data is stored in an image memory 17. Thesecond image data D14 is subjected to a predetermined processing by asignal processing circuit (not shown), and then, one frame's worth (orone field's worth) of image data is stored in an image memory 18.

The first image data D10 read out from the image memory 17 is subjectedto data compression in a two-dimensional DCT (discrete cosinetransformation) processing circuit 21, a quantization processing circuit22 and a Huffman encoding processing circuit 23.

The first image data D10 is divided into a plurality of blocks andoutputted to the DCT processing circuit 21 for each block. Each block isconstituted by 8×8 pixels. The image data Pxy of this block consistingof 8×8 pixels is subjected to two-dimensional DCT in the DCT processingcircuit 21 and transformed to a DCT coefficient. Namely, in the presentembodiment, DCT is adopted as the orthogonal transform of the imagedata.

The DCT coefficient outputted from the DCT processing circuit 21 isinputted to the quantization processing circuit 22 for each 8×8 pixelblock. The DCT coefficient is quantized using a predeterminedquantization table in the quantization processing circuit 22, and thus,a quantized DCT coefficient is obtained.

The quantization processing circuit 22 is connected to the Huffmanencoding processing circuit 23 and an inverse quantization processingcircuit 24. The quantized DCT coefficient is Huffman-encoded by theHuffman encoding processing circuit 23 and converted to code words andthen recorded in to a recording medium (such as, for example, an IC cardmemory) M as encoded data A. On the other hand, the quantized DCTcoefficient is subjected to an inverse quantization in the inversequantization processing circuit 24 by using the quantization table usedin the quantization processing circuit 22, and thus the DCT coefficientis restored.

The restored DCT coefficient obtained by the inverse quantizationprocessing circuit 24 is inputted to an IDCT processing circuit 25, inwhich an IDCT (inverse discrete cosine transformation) is carried outwith respect to the restored DCT coefficient, whereby the first imagedata D10' is restored for each block consisting of 8×8 pixels.

The IDCT processing circuit 25 is connected to an expansion processingcircuit 26. In the expansion processing circuit 26, an expansion processis carried out according to a hierarchical procedure of the JPEG (JointPhotographic Experts Group) with respect to the restored first imagedata D10'. In the expansion process in the embodiment, an upsampling isperformed in a horizontal direction of the restored first image dataD10' in such a manner that the number of pixels arranged in a horizontaldirection is doubled, since the CCDs 12 and 14 are arranged in such amanner that the pixels of the CCDs 12 and 14 are offset to each other byhalf of the pixel pitch. Namely, restored expanded image data D12 isgenerated by the expansion process.

The IDCT processing circuit 25 is also connected to an expansionprocessing circuit 27 which is connected to the image memory 18. In theexpansion processing circuit 27, third image data D16 having a secondresolution higher than the first resolution is generated based on therestored first image data D10' inputted from the IDCT 25 and the secondimage data D14 inputted from the image memory 18. The expansionprocessing circuit 27 is connected to a subtractor 31 which is connectedto the expansion processing circuit 26. In the subtractor 31, theexpanded image data D12 inputted from the expansion processing circuit26 is subtracted from the third image data D16 inputted from theexpansion processing circuit 27, whereby supplementary data D18 isgenerated. The subtraction in the subtractor 31 is performed withrespect to each corresponding pixel of the third image data D16 and theexpanded image data D12.

The supplementary data D18 obtained by the subtractor 31 is inputted toa DCT processing circuit 32, in which a two-dimensional DCT is performedon the supplementary data D18 so that a DCT coefficient of thesupplementary data D18 is obtained. The DCT coefficient of thesupplementary data D18 is quantized in a quantization processing circuit33, which is connected to a Huffmam encoding processing circuit 34. Thequantized DCT coefficient of the supplementary data D18 isHuffman-encoded at the Huffman encoding processing circuit 34 andconverted to code words and then recorded to the recording medium M asencoded data B'.

FIG. 2 is a block diagram of a reproducing system of the electronicstill camera having the recording system shown in FIG. 1.

The encoded data A and B' read out from the recording medium M areinputted into Huffman decoding processing circuits 41 and 42, in whichquantized DCT coefficients are decoded based on the encoded data A andB', respectively.

The quantized DCT coefficient obtained by the Huffman decodingprocessing circuit 41 is subjected to an inverse quantization in aninverse quantization processing circuit 43, whereby the DCT coefficientis restored. The restored DCT coefficient is inputted to an IDCTprocessing circuit 44 in which an inverse discrete cosine transformationis carried oat with respect to the restored DCT coefficient, whereby thefirst image data D10' is restored.

Similarly, the quantized DCT coefficient obtained by the Huffmandecoding processing circuit 42 is restored to the DCT coefficient by aninverse quantization processing circuit 45, and the restored DCTcoefficient is subjected to an inverse discrete cosine transformation inan IDCT processing circuit 46, whereby supplementary data D18' isrestored.

The restored first image data D10' is sequentially written to an imagememory 51 and accumulated therein. The image memory 51 is connected to acommon terminal 52c of a switch 52 which has a first terminal 52aconnecting to a data input terminal of a D/A convertor 53 and a secondterminal 52b connecting to an expansion processing circuit 54.

Switch 52 is swithced over by a control circuit 59 having a microcomputer, according to recognition information described later, fromwhich it is determined whether the encoded data B' is recorded to therecording medium M.

The D/A convertor 53 is connected to an output device 55, such as adisplay device, through an analog image signal output terminal providedon an outer surface of the housing of the electronic still camera.Therefore, when switch 52 is switched to the first terminal 52a, therestored first image data D10' read from the image memory 51 isconverted to an analog signal by the D/A convertor 53, so that thereproduced image is shown on the output device 55 at the firstresolution, such as the NTSC system.

When the switch 52 is switched to the second terminal 52b, the restoredfirst image data D10' read from the image memory 51 is inputted toexpansion processing circuit 54, in which expansion process is carriedout according to the hierarchical procedure of the JPEG with respect tothe restored first image data D10', similar to the expansion processingcircuit 26. Namely, each pixel of the restored first image signal D10'is upsampled by two times in a horizontal direction, so that restoredexpanded image data D12 is generated.

The expansion processing circuit 54 and the IDCT processing circuit 46are connected to an adder 55 so that the restored expanded image dataD12 and the supplementary data D18' are inputted to the adder 55,respectively. In the adder 55, a pixel of the restored expanded imagedata D12 and the corresponding pixel of the supplementary data D18' areadded to each other, and thus a third image data D16' having a higherresolution than the first resolution is restored. An output terminal ofthe adder 55 is connected to an image memory 56, so that the third imagedata D16' of high resolution is stored to the image memory 56.

One frame's worth (or one field's worth) of the high resolution imagedata D16' is read out from the image memory 56 and converted to ananalog image signal of high resolution by a D/A convertor 57 which isconnected to a high resolution image signal output device 58 through ananalog image signal output terminal provided on an outer surface of thehousing of the electronic still camera. Namely, the image is shown onthe output device 58 at a second resolution higher than that of theoutput device 55.

Handling of data in each of the circuits shown in FIGS. 1 and 2 isdescribed below.

FIG. 3 shows an example of the first image data D10 divided into a blockconsisting of 8×8 pixels. In the DCT processing circuit 21, the firstimage data D10 inputted thereto is subjected to the two-dimensional DCTtransformation, and thus the DCT coefficient is obtained for each block,as shown in FIG. 3.

The two-dimensional DCT transformation is performed according to thefollowing equation (1): ##EQU1## wherein x,y=position of pixel in block

u,v=position of DCT coefficient ##EQU2## Ls=128; bit precision of Pxy=8bit

Note, in the equation (1), Pxy indicates a value of each pixel of firstimage data D10, and is image data consisting of luminance signals of 256gradations (8 bit precision). The DCT coefficient obtained by the DCTprocessing circuit 21 shows an amplitude spectral density. In case of ablock consisting of 8×8 pixels, 64 (=8×8) of the DCT coefficient areobtained by the two-dimensional DCT transformation according to theequation (1). Among the 64 DCT coefficients, coefficient S₀₀, located atthe element position (0,0), is a DC (Direct Current) component, and theremaining 63 coefficients are AC (Alternating Current) components. TheAC components show how many higher frequency components exist in theoriginal image as the coefficient is changed from S₁₀ to S₇₇ (=amplitudespectral density). Namely, the coefficient S₇₇ expresses the coefficienthaving the highest spatial frequency. The DC component shows a meanvalue of whole block of 8×8 pixels.

Note, in equation (1), Ls is subtracted from each pixel value Pxy, sothat the expected value of the DC component becomes close to 0. Due tothis, the DC componet can be encoded to a code word having a shorterlength in the Huffman encoding, so that an efficiency of datacompression of the image data is improved.

The DCT coefficients are inputted to the quantization processing circuit22 in which a quantization table Quv, as shown in FIG. 3, is provided,and thus each of the DCT coefficients is quantized using thequantization table Quv.

The quantization is performed according to the following equation:

    Ruv=round (Suv/quv) {0≦u, v≦7}

The term "ground" in this equation means rounding to the closestinteger. Namely, the quantized DCT coefficient Ruv as shown in FIG. 3 isobtained by division and rounding off between the respective elements ofthe DCT coefficients Suv and the respective elements of the quantizationtable Quv. The quantized DCT coefficient Ruv is inputted to the Huffmanencoding processing circuit 23 and the Huffman encoding processingcircuit 34.

The operation of Huffman encoding the quantized DCT coefficient Ruv willbe explained referring to FIG. 4 to FIG. 8.

The encoding method differs between the DC component R₀₀ and ACcomponent (the quantized DCT coefficient Ruv other than the DC componentR₀₀). The encoding of the DC component R₀₀ is carried out as follows:

First, a difference value between the quantized DCT coefficient R₀₀ ofthe block to be encoded and the quantized DCT coefficient R₀₀ of thepreceding encoded block is found. It is decided which of the groupsshown in FIG. 4 this difference value belongs to. The code wordexpressing the number of that group is found from the code table (codingtable of DC component) shown in FIG. 5. For example, when the quantizedDCT coefficient R₀₀ of the block to be encoded is 16 and the quantizedDCT coefficient R₀₀ of the preceding encoded block is 25, the differencevalue is -9, and therefore it is decided that group number (SSSS) of thegroup to which the difference value=-9 belongs is "4" from the groupnumber table of FIG. 4. Further, it is decided that the code word ofthat group number (SSSS) is "101" from the code table of FIG. 5.

Subsequently, the order of the difference value in that group in thegroup number table of FIG. 4 is expressed by an additional bit. Forexample, the difference value=-9 is seventh in order from the smallest(i.e., -15, -14, -13, -12, -11, -10, -9 equals seventh from the smallest(-15) in the group of the group number (SSSS)=4, and therefore theadditional bit becomes "0110" (base 2 notation, where the firstdifference is equal to 0000; the second difference is equal to 0001; thethird difference is equal to 0010; the fourth difference is equal to0011; the fifth difference is equal to 0100; the sixth difference isequal to 0101; and the seventh difference is equal to 0110). Namely, theHuffman encoded word for the quantization DC component R(Y)₀₀ of theblock which is now being encoded becomes "1010110".

On the other hand, the encoding of the AC component of the quantized DCTcoefficient is performed by the processing routine shown in FIG. 6. InStep 120, 63 quantized DCT coefficients are subjected to a zigzagscanning in the order shown in FIG. 7 and rearranged into aone-dimensional array of data. Then, it is decided in Step 122 whetherthe respective quantized DCT coefficient values arranged in onedimension are "0". When any quantized DCT coefficient is "0", Step 124is executed so that the quantized DCT coefficients which are "0" arecounted. By this, the length of the continuous "0"s, that is, the runlength (NNNN) is obtained.

Contrary to this, when it is determined in Step 122 that the quantizedDCT coefficient is not "0", in Step 126, the group classification,similar to that for the DC component, is carried out and, at the sametime, the additional bit is obtained. The group classification of thequantized DCT coefficient of the AC component differs from the groupclassification of the DC component and is carried out for the quantizedDCT coefficient thereof per se. Namely, when the quantized DCTcoefficient is for example "4", a group number (SSSS) "3" is obtained byreferring to a table in the same way as in FIG. 4. Since the quantizedDCT coefficient "4" exists at fifth place from the smallest in the groupof the group number (SSSS)=3 (i.e., -7, -6, -5, -4, 4 equals fifth placeusing a table in the same way as in FIG. 4), the additional bit becomes"100".

Subsequently, in Step 130, AC code table (not shown) is referred to,and, for example, where the run length of data immediately before thequantized DCT coefficient of "4" is "0", the code word "100" is obtainedbased on this run length and the group number (SSSS)=3. Then, bycombining this code word "100" and the additional bit "100" obtained inStep 126, the two-dimensional Huffman encoding word "100100" isobtained.

The result of performing the Huffman encoding for the quantized DCTcoefficient of FIG. 3 is indicated as the encoded data 100 of FIG. 8.The encoded data 100 are sequentially recorded on the IC memory card M.

The quantized DCT coefficient outputted from the quantization processingcircuit 22 is also inputted to the inverse quantization processingcircuit 24, so that the quantized DCT coefficient is restored to the DCTcoefficient by using the quantization table Quv used in the quantizationprocessing circuit 22. FIG. 9 shows the restored DCT coefficient Suvwhich is obtained by performing inverse-quantization with respect to thequantized DCT coefficient Ruv shown in FIG. 3. The restored DCTcoefficient Suv is inputted to the IDCT processing circuit 16, in whichthe restored DCT coefficient Suv is subjected to two-dimensional IDCTtransformation, so that the first image data D10' is restored. Thetwo-dimensional IDCT transformation is performed according to thefollowing equation (2): ##EQU3## wherein x,y=position of pixel in block

u,v=position of DCT coefficient ##EQU4## Ls=128; bit precision of Pxy=8bit

The first image data D10' restored by performing the two-dimensionalIDCT transformation with respect to the DCT coefficient Suv is shown inFIG. 9.

The restored first image data D10' is inputted into the expansionprocessing circuit 26, in which the expansion process according to thehierarchical procedure of the JPEG algorithm is performed so that thenumber of pixels of the restored first image data D10' is doubled in ahorizontal direction of the original image data generated by the CCD 12or 14. FIG. 10 shows an example of the first image data D10' restored bythe IDCT processing circuit 25 or 44, and an example of the restoredexpanded image data D12 generated by the expansion processing circuit 26or 54. As shown in FIG. 10, the upsampling in the horizontal directionaccording to the hierarchical procedure is performed in such a mannerthat, in the expanded image data D12, a value of a pixel is defined as amean value of two pixels adjacent thereto.

The first image data D10, which is generated by the CCD 12 and A/Dconverted by the A/D convertor 15, and the second image data D14, whichis generated by the CCD 14 and A/D converted by the A/D convertor 16,are offset to each other by half of a distance between the centers oftwo adjacent pixels in a horizontal direction of the image data D10 andD12 as shown by reference T in FIG. 10. The second image data D10 andthe restored first image data D10' outputted from the IDCT processingcircuit 25 are inputted to the expansion processing circuit 27, in whichthe first and second image data D10' and D14 are arranged in thehorizontal direction in such a manner that a pixel of the restored firstimage data D10' and a pixel of the second image data D14 are alternatelydisposed in a horizontal direction, and thus, the third image data D16is generated as shown in FIG. 10.

In the subtractor 31, a subtraction is performed with respect to eachcorresponding pixel of the expanded image data D12 and the third imagedata D16, and thus, the supplementary data D18 for reproducing a highresolution image data is generated as shown in FIG. 10 and inputted tothe DCT processing circuit 32.

In the DCT processing circuit 32, the supplementary data D18 istransformed to the DCT coefficients, which are inputted to thequantization processing circuit 33 in which each of the DCT coefficientsis quantized by using a quantization table different from thequantization table Quv used in the quantization processing circuit 22.The reason why these quantization tables are different from each otheris that a distribution of values of the DCT coefficients, i.e.,statistical characteristics of the DCT coefficients are different fromeach other. Namely, the step width for quantization in the quantizationtable used in the quantization processing circuit 22 may be changed inaccordance with the original image data outputted from the CCD 12.

The quantized DCT coefficient obtained by the quantization processingcircuit 33 is inputted to the Huffman encoding processing circuit 34, inwhich the Huffman encoding is performed as in the Huffman encodingprocessing circuit 23, and thus, the encoded data B' is generated. Note,a Huffman table used in the Huffman encoding processing circuit 34 isdifferent from a Huffman table used in the Huffman encoding processingcircuit 23. The reason is described below.

The quantized DCT coefficient inputted to the Huffman encodingprocessing circuit 23 is obtained based on the first image data D10which corresponds to the original image, and the quantized DCTcoefficient inputted to the Huffman encoding processing circuit 34 isobtained based on the supplementary data D18 for reproducing a highresolution image. The first image data D10 has different statisticalcharacteristics from that of the supplementary data D18. Therefore,there is a difference in the statistical characteristics between the twoquantized DCT coefficients, and thus, the Huffman tables used in theHuffman encoding processing circuits 23 and 34 are different.

The encoded data B' is recorded on the IC memory card M with the encodeddata A. Note, when the encoded data A and B' are recorded to the ICmemory card M, recognition information is also recorded to the IC memorycard M, so that it is determined whether the data read out from the ICmemory card M is the encoded data A or the encoded data B'.

A process in which the encoded data A and/or B' are read from the ICmemory card M to reproduce the image data is described below withreference to FIGS. 2 and 9. Note, when high resolution image data isreproduced, both of the encoded data A and B' are read from the ICmemory card M. Conversely, when a normal resolution image data isreproduced, the encoded data B' need not be read from the IC memory cardM.

The encoded data A and B' are read out from the IC memory card Maccording to the recognition information recorded when the data A and B'are recorded on the IC memory card M, and inputted to the Huffmandecoding processing circuits 41 and 42, respectively.

In the Huffman decoding processing circuit 41, the quantized DCTcoefficient Ruv is decoded by using the Huffman table used in theHuffman encoding processing circuit 23. FIG. 9 shows the quantized DCTcoefficient Ruv obtained by decoding the encoded data 100 shown in FIG.8, which is an example of the encoded data A read from the IC memorycard M.

The quantized DCT coefficient is inputted to the inverse quantizationprocessing circuit 43, in which the inverse-quantization is performed byusing a quantization table used in the quantization processing circuit22. Namely, the quantization table Quv is multiplied by each of therestored quantization DCT coefficients Ruv for each element, and thus,the DCT coefficient Suv is restored as shown in FIG. 9. Note, thequantization table used in the IDCT processing circuit 43 is the same asthe quantization table used in the inverse quantization processingcircuit 24.

The DCT coefficient restored by the inverse quantization processingcircuit 43 is inputted to the IDCT processing circuit 44, in which theDCT coefficient is subjected to a two-dimensional IDCT transformation,and thus, the restored image data D10' (Pxy) as shown in FIG. 9 isrestored. This restored image data D10' is substantially the same as therestored image data obtained by the IDCT processing circuit 25.Therefore, the inverse quantization circuit 43 can serve as the inversequantization circuit 24, and the IDCT processing circuit 44 can beserved as the IDCT processing circuit 25. In the embodiment, these areseparate circuits for convenience of explanation.

The restored image data D10' obtained by the IDCT processing circuit 44is inputted to the image memory 51. When storing of one frame's worth(or one field's worth) of the image data D10' in the image memory 51 hasbeen completed, the restored image data D10' is transferred from theimage memory 51 to the D/A convertor 53 or the expansion processingcircuit 54, depending upon the state of the switch 52.

When the restored image data D10' is inputted to the D/A converter 53,the restored image data D10' is converted to analog data and outputtedto the output device 55, so that the image having a normal resolution(the first resolution) is shown on the output device 55. Conversely,when the restored image data D10' is inputted to the expansionprocessing circuit 54, each pixel of the restored image data D10' isexpanded in a horizontal direction based on the hierarchical procedureof the JPEG algorithm, and the restored image signal D12 is obtained asshown in FIG. 10. The expanded restored image signal D12 is inputted tothe adder 55.

On the other hand, the encoded data B' read out from the IC memory cardM is decoded by the Huffman decoding processing circuit 42. The Huffmantable used in the Huffman decoding processing circuit 42 is the same asthat used in the Huffman encoding processing circuit 34. The restoredquantized DCT coefficient obtained by the Huffman decoding processingcircuit 42 is subjected to an inverse quantization in the inversequantization processing circuit 45, and thus, the DCT coefficient isrestored. The quantization table used in the inverse quantizationprocessing circuit 45 is the same as that used in the quantizationprocessing circuit 33. The restored DCT coefficient is subjected to atwo-dimensional IDCT transformation in the IDCT processing circuit 46,and thus, the supplementary data D18' is restored, an example of whichis shown in FIG. 10. Note, the supplementary data D18' is substantiallythe same as the supplementary data D18 outputted from the subtractor 31.

The restored supplementary data D18' obtained by the IDCT processingcircuit 46 is also inputted to the adder 55, to which the restoredexpanded image data D12 is inputted from the expansion processingcircuit 54. In the adder 55, the supplementary data D18' is added to therestored expanded image data D12 with respect to each of thecorresponding components positioned at the same pixel position, and thusthe third image data D16' having a high resolution is reproduced. Anexample of the third image data D16' is shown in FIG. 10.

The third image data D16' is inputted to the image memory 56. When thestoring of one frame's worth (or one field's worth) of the third imagedata D16' in the image memory 56 has been completed, the third imagedata D16' is sequentially scanned and read out from the image memory 56,and inputted to the D/A convertor 57. The third image data D16' isconverted to an analog image signal and outputted to the high resolutionimage signal output device 58, and thus, an image having a higherresolution in a horizontal direction than that of an image indicated bythe output device 55 is shown on the high resolution image signal outputdevice 58.

Thus, in this embodiment, the high resolution image is reproduced byusing the first image data D10 generated in the CCD 12 and the secondimage data D12 generated in the CCD 14, in which the same object imageis formed as the CCD 12, and which is provided in such a manner that thecorresponding pixels of the CCDs 12 and 14 are offset by a half ofdistance between the centers of two adjacent pixels.

If a CCD outputting a high resolution image signal is provided, i.e., ifa CCD in which a pitch between two adjacent pixels is narrower thanthose of the CCDs 12 and 14 is provided, two CCDs need not be provided.

Namely, when such a high resolution CCD is provided, the high resolutionimage signal C outputted from the CCD is subjected to downsampling, sothat a normal resolution image data A is obtained. The normal resolutionimage data A is transformed to a quantized DCT coefficient by the DCTprocessing circuit 21 and the quantization processing circuit 22. Then,the quantized DCT coefficient is transformed to a normal resolutionimage data A' by the inverse quantization circuit 24 and the IDCTprocessing circuit 25, and further, the normal resolution image data A'is subjected to an expansion procedure by the expansion processingcircuit 26 to generate a restored expanded image data D12. Then,supplementary data D18 is generated based on the restored expanded imagedata D12 and the high resolution image signal C, and the supplementarydata D18 and the normal resolution image signal A are subjected to datacompression, such as a DCT transformation, a quantization and a Huffmanencoding process, and recorded to the IC memory card M.

A construction which includes a high resolution CCD does not need theCCD 14, the half-mirror 13, the A/D convertor 16 and the expansionprocessing circuit 27.

Note, in comparing the embodiment of the device shown in FIGS. 1 and 2with a device in which a normal resolution image data is generated bysubsampling the high resolution image data obtained by a CCD outputtingthe high resolution image, and the supplementary data is obtained basedon the normal resolution image data and the high resolution image data,the statistical characteristics of the supplementary data are differentin each device.

On the other hand, although the embodiment shown in FIGS. 1 and 2 is anelectronic still camera which can record and reproduce both a normalresolution image and a high resolution image, the present invention canbe applied to a record-reproduction device in which high resolutionimage data is inputted from another device, and which can output bothnormal resolution image data and high resolution image data.

Namely, in this record-reproduction device, an image data generatingdevice including a CCD, an A/D converter and the like is not provided,and the normal resolution image data is generated based on the highresolution image data. The difference data (i.e., the supplementarydata) between the high resolution image data and the normal resolutionimage data is recorded in a recording medium so that both the normalresolution image data and the high resolution image data can bereproduced.

The supplementary data D18 is used whew the high resolution image dataD10 is generated based on the normal resolution image data D10 isrecorded in a small recording area of the recording medium M. The reasonwhy the size of the recording area is small is as follows:

The first image data D10 obtained by the CCD 12 and the second imagedata D14 obtained by the CCD 14 represent the same object, and theexpanded image data D12 is generated in accordance with the first imagedata D10 while the high resolution image data D16 is generated inaccordance with the first and second image data D10 and D14. Therefore,the amplitude values of the supplementary data D18 obtained bysubtracting the expanded image data D12 from the high resolution imagedata D16 concentrates on the value "0". Accordingly, the entropy of thesupplementary data D18 is small. Therefore, the supplementary data D18can be encoded with a short code word, and thus, the volume of recordingarea in which the encoded word corresponding to the supplementary datais recorded is small. Namely, the amount of recorded data in therecording medium is drastically reduced in comparison with a device inwhich both of the normal resolution image data D10 and the highresolution image data D16 are recorded in the recording medium.

On the other hands the JPEG algorithm includes a hierarchical process inwhich a progressive build-up indication is carried out. In thishierarchical process, the original image is subjected to downsampling,so that images having a size of 1/2, 1/4 . . . 1/2^(n) of the originalimage are generated. The image having the smallest size (1/2^(n) of theoriginal image) is subjected to a two-dimensional DCT transformation,quantization and Huffman encoding procedure (a DCT system), and thus,encoded data is generated. Then, the encoded data is decoded to generatea restored image having the size of 1/2^(n) of the original image. Then,the restored image is expanded by two times, and thus, the image ischanged to an image having a size of 1/2^(n-1) of the original image.Supplementary data K(n-1) is obtained based on the image of 1/2^(n-1)size and another image which is generated by downsampling the originalimage and has a size of of 1/2^(n-1) of the original image. Thesupplementary data K(n-1) is transformed to encoded data according tothe DCT system.

Such an encoding procedure is repeatedly performed until an image havingthe same size as the original image is produced, and the encoded datahaving the smallest size image (corresponding to the encoded data of abasic image or the first image data of the embodiment) is transmitted toa receiving device, in which the basic image is restored based on theencoded data and shown on a display, and using the supplementary dataK(p-1) (p=n, n-1, n-2, . . . 1) if necessary, an image having a size of1/2^(p-1) of the original image is restored based on the restored basicimage (corresponding to the high resolution image of the embodiment). Byrepeating such a procedure, an image resolution is increased step bystep, and this is called a progressive build-up indication.

In the embodiment, the high resolution image data D16 is obtained basedon the first image data D10, which is the basic image, and the secondimage data D14. The first image data D10 is subjected to the DCTtransformation and quantization to transform it to the quantized DCTcoefficient, and then the quantized DCT coefficient is restored to imagedata D10'. The image data D10' is subjected to an expansion processaccording to the hierarchical process, and thus, the expanded image dataD12 is obtained. The supplementary data D18 is the difference betweenthe high resolution image data D16 and the supplementary data D12. Thefirst image data D10 and the supplementary data D18 are DCT-transformedand Huffman-encoded to be recorded to the IC memory card M. Theseprocesses of the embodiment is similar to the hierarchical process inwhich the progressive built-up indication is carried out. Therefore, inthe embodiment, the encoded data can be recorded to the IC memory card Maccording to a communication protocol of the hierarchical process.

Namely, in the hierarchical process of the JPEG algorithm, various kindsof information are added to the encoded data so that the image can beproperly restored in the receiving device, and such added informationnecessary for restoring operation is added to the encoded data A and B'in the embodiment, and recorded to the IC memory card M. The addedinformation includes the quantization table and the Huffman table usedin the encoding process. The data format in the hierarchical process isas follows: Namely, one frame in which the encoded data obtained byencoding one frame of image data and one supplementary data issandwiched by a "SOI" (Start of Image) marker and an "EOI" (End ofImage) marker as a basic unit, and the encoded data corresponding to thesupplementary data obtained for each step is provided in the frame. Suchframes are continuousely arranged. A frame header indicating parametersrelating the frame is added in front of the frame.

Therefore, by recording the encoded data A and B' to the IC memory cardM according to the hierarchical process, a decoder for the JPEGalgorithm can restore the image from data read out from the IC memorycard M. Namely, the data recorded to the IC memory card M by theelectronic still camera of the embodiment can be transmitted, withoutmodifying the data, to a receiving device by using a communicationprotocol according to the JPEG algorithm.

Note, since a color image can be encoded in the JPEG algorithm, eachcolor component can be encoded and recorded to the IC memory card M, ina data format according to the hierarchical process, in the embodiment.

Although two images, i.e., the normal resolution image and the highresolution image are recorded to or reproduced from the IC memory card Min the embodiment, further supplementary data may be generated, encodedand recorded to the IC memory card M so that another image, of a higherresolution than the high resolution image, can be reproduced.

FIG. 11 is a block diagram of a recording system of an electronic stillcamera to which a second embodiment of the present invention is applied.The second embodiment has a construction in which further complementarydata is generated and recorded to the IC memory card M in the form of anencoded data, so that a higher resolution image than that in the firstembodiment is reproduced. Note, in FIG. 11, the same components as inthe first embodiment are indicated by the same reference numbers, and adetailed explanation thereof is omitted.

A half-mirror 71 is provided between the focus lens 11 and thehalf-mirror 13, so that a part of the light from the object is fed tothe half-mirror 13, and the remaining light from the object is fed to ahalf-mirror 72. The light passing through the half-mirror 72 enters aCCD 73, and the light reflected on the half-mirror 72 enters a CCD 74.Thus, the same image is formed on each of the CCDs 73 and 74 as the CCDs12 and 14.

The CCDs 73 and 74 are arranged in such a manner that a pixel of theelectric conversion element of the CCD 73 and a pixel of the CCD 74 areoffset to each other by half the distance between the centers of twoadjacent pixels in both a horizontal direction and a vertical direction,respectively, of the image formed on the CCDs 73 and 74.

Image data generated by the CCD 73 is converted to digital image data byan A/D converter 75, and stored in an image memory 76. Similarly, imagedata generated by the CCD 74 is converted to digital image data by anA/D converter 77, and stored in an image memory 78. One frame's worth(or one field's worth) of image data is read from the image memories 76and 78, respectively, and inputted into an expansion processing circuit79. In the expansion processing circuit 79, an upsampling is performedin a horizontal direction of the image data in such a manner that thenumber of pixels arranged in a horizontal direction is doubled, so thata high resolution image data is obtained. The high resolution image dataD16 outputted from the expansion processing circuit 27 is also inputtedto the expansion processing circuit 79.

In the expansion processing circuit 79, the the high resolution imagedata D16 and the high resolution image data obtained based on the imagedata outputted from the memories 76 and 78 are subjected to an expansionprocess, so that the number of pixels arranged in a vertical directionis doubled, and thus a second high resolution image data D20 isgenerated in the expansion processing circuit 79.

On the other hand, the quantized DCT coefficient of the supplementarydata outputted from the quantization circuit 33 is subjected to aninverse quantization in an inverse quantization circuit 81, and thus,the quantized DCT coefficient is converted to a DCT coefficient. The DCTcoefficient is subjected to a two-dimentional IDCT in an IDCT processingcircuit 82, and thus, the supplementary data D21 is restored. In anadder 83, the restored supplementary data D21 is added to the expandedimage data D12 inputted from the expansion processing circuit 26, andthus, high resolution image data D22 is restored. This restored highresolution image data D22 is subjected to an expansion process in anexpansion processing circuit 84, so that the number of pixels arrangedin a vertical direction is doubled, and thus, a third high resolutionimage data D24 is generated.

In a subtractor 85, the third high resolution image data D24 issubtracted from the second high resolution image data D20, and thus,second supplementary data D26 is obtained. The second supplementary dataD26 is subjected to the DCT transformation in a DCT processing circuit87, and then subjected to the quantization in a quantization processingcircuit 88. The quantized DCT coefficient obtained by the quantizationprocessing circuit 88 is Huffman-encoded in a Huffman encodingprocessing circuit 89, and recorded to the IC memory card M as theencoded data CD'.

Based on the image data A and the supplementary data B' and CD' readfrom the IC memory card M, an image in which the resolution is increasedby two times in both the horizontal direction and the vertical directionis obtained. Namely, a higher resolution image than that in the firstembodiment is reproduced. Thus, by increasing the number of the CCDs bytwo times, the resolution of the image is improved.

FIG. 12 shows upsampling in a horizontal direction and a verticaldirection. In the example, first, image data is subjected to upsamplingso that the number of pixels is doubled in the horizontal direction, andthus, image data A is obtained. Then, the image data A expanded in thehorizontal direction is subjected to upsampling so that the number ofpixels is doubled in the vertical direction, and thus, image data A' isobtained.

In the other example in FIG. 12, image data Axy obtained by the CCD 12and image data Bxy obtained by the CCD 14 are subjected to upsampling insuch a manner that each of the data αxy and βxy is obtained in anarbitrary manner, in which a mean value of adjacent Axy and Bxy data iscalculated, for example.

Note, for transforming the original image data from a spatial coordinateaxis to a spatial frequency axis, it is possible to use other orthogonaltransformations such as Fourier transformation, Hadamard'stransformation, Harr transformation, etc. in place of the discretecosine transformation.

Further, as for the orthogonal transformation and the encoding procedurefor quantizing orthogonal transformation coefficients, although theHuffman encoding is used in the above embodiment, another entropyencoding method such as an arithmetic encoding method can be used.

Furthermore, the size of the block of the first image data D10 is notrestricted to 8×8 pixels.

Although the embodiments of the present invention have been describedherein with reference to the accompanying drawings, obviously manymodifications and changes may be made by those skilled in this artwithout departing from the scope of the invention.

The present disclosure relates to subject matter contained in JapanesePatent Application No. 5-197993 (filed on Jul. 15, 1993) which isexpressly incorporated herein, by reference, in its entirety.

I claim:
 1. A device for processing an image signal, comprising:firstmeans for generating first image data based on an optical image, saidfirst image data having a first resolution; second means for generatingsecond image data based on said optical image, said second image databeing offset from said first image data by a predetermined amount on aspatial coordinate axis; third means for generating third image databased on said first and second image data, said third image data havinga second resolution higher than said first resolution; means fortransforming said first image data to an expanded image data having anumber of pixels equal to a number of pixels of said third image data;means for subtracting said expanded image data from said third imagedata to generate supplementary data; means for applying data compressionto said first image data and said supplementary data; and means forrecording said first image data and said supplementary data compressedby said data compression means to a recording medium.
 2. A deviceaccording to claim 1, wherein said data compression applying meanscomprises:means for applying an orthogonal transformation to said firstimage data and said supplementary data to obtain orthogonaltransformation coefficients of said first image data and saidsupplementary data; means for quantizing said orthogonal transformationcoefficients to obtain quantized orthogonal transformation coefficients;and means for encoding said quantized orthogonal transformationcoefficients to obtain encoded data.
 3. A device according to claim 2,wherein said orthogonal transformation applied by said orthogonaltransformation applying means comprises a discrete cosinetransformation, and said encoding performed by said encoding meanscomprises a Huffman encoding.
 4. A device according to claim 2, furthercomprising:means for applying an inverse-quantization to said orthogonaltransformation coefficients of said first image data; and means forapplying an inverse-orthogonal transformation to said orthogonaltransformation coefficients of said first image data to obtain restoredimage data; said third generating means generating said third image databased on said restored image data and said second image data.
 5. Adevice according to claim 2, further comprising:means for applying aninverse-quantization to said orthogonal transformation coefficient ofsaid first image data; and means for applying an inverse-orthogonaltransformation to said orthogonal transformation coefficient of saidfirst image data to obtain restored image data; said transforming meanstransforming said restored image data to said expanded image data.
 6. Adevice according to claim 2, further comprising:means for decoding saidencoded data to obtain restored quantized orthogonal transformationcoefficients of said first image data and said supplementary data; meansfor inverse-quantizing said restored quantized orthogonal transformationcoefficients to obtain restored orthogonal transformation coefficientsof said first image data and said supplementary data; means for applyingan inverse-orthogonal transformation to said restored orthogonaltransformation coefficients to restore said first image data and saidsupplementary data; means for outputting said restored first image datato a first display device showing an image at said first resolution;means for transforming said restored first image data to restoredexpanded image data having a number of pixels equal to a number ofpixels of said third image data; means for reproducing said third imagedata based on said restored expanded image data and said restoredsupplementary data; and means for outputting said restored third imagedata to a second display device showing an image at said secondresolution.
 7. A device according to claim 1, wherein said first andsecond generating means comprise charge coupled devices on which opticalimages are formed, respectively, said second generating means beingoffset from said first generating means by half a distance between thecenters of two adjacent pixels in a horizontal direction of said opticalimage, and said third generating means arranging said first and secondimage data in a horizontal direction in a predetermined manner togenerate said third image data.
 8. A device according to claim 1,wherein said transforming means performs an upsampling on said spatialcoordinate axis in such a manner that, in said expanded image data, thevalue of a pixel is defined as a mean value of two adjacent pixels.
 9. Adevice according to claim 1, wherein said subtracting means performs asubtraction with respect to each corresponding pixel of said expandedimage data and said third image data.
 10. A device according to claim 1,wherein said first and second generating means comprise charge coupleddevices on which optical images are formed, respectively, said secondgenerating means being offset from said first generating means by half adistance between the centers of two adjacent pixels in a verticaldirection of said optical image, said third generating means arrangingsaid first and second image data in said vertical direction in apredetermined manner to generate said third image data.
 11. A deviceaccording to claim 1, further comprising:means for reading said firstimage data and said supplementary data recorded to said recording mediumin such a manner that said first image data and said supplementary dataare expanded and restored; means for outputting said restored firstimage data to a display device showing an image at said firstresolution; means for transforming said restored first image data torestored expanded image data having a number of pixels equal to saidthird image data; and means for reproducing said third image data basedon said restored expanded image data and said restored supplementarydata.
 12. A device according to claim 1, wherein said image signalprocessing device is provided in an electronic still camera.
 13. Adevice for processing an image signal, comprising:an image sensor forgenerating an image signal corresponding to a subject, said image sensoroutputting image data corresponding to said subject; an A/D converterfor A/D converting said image data to generate a first image data andsecond image data, each of which has a first resolution, and which areoffset to each other by a predetermined amount in a spatial coordinateaxis of said original image data; a first expansion circuit forgenerating third image data based on said first and second image data,said third image data having a second resolution higher than said firstresolution; a second expansion circuit for performing an expansionprocess to transform said first image data to expanded image data havinga number of pixels equal to said third image data; a subtractor forsubstracting said expanded image data from said third image data togenerate supplementary data; a data compression circuit for applyingdata compression to said first image data and said supplementary data;and a recording medium to which said first image data and saidsupplementary data compressed by said recording means are recorded. 14.A device according to claim 13, further comprising:a data expansioncircuit for reading said first image data and said supplementary datarecorded to said memory in such a manner that said first image data andsaid supplementary data are expanded and restored; a first D/A converterfor D/A converting said restored first image data to analog first imagedata and outputting said analog first image data to a display deviceshowing an image at said first resolution; a third expansion circuit forperforming an expansion process to transform said restored first imagedata to restored expanded image data having a number of pixels equal tosaid third image data; an adder for adding said restored supplementarydata to said restored expanded image data to reproduce said third imagedata; a second D/A converter for D/A converting said reproduced thirdimage data and outputting said analog third image data to a displaydevice showing an image at said second resolution.
 15. A device forprocessing an image signal including first image data and supplementarydata, which are subjected to an orthogonal transformation, aquantization and an encoding, and recorded to a recording medium, saidfirst image data having a first resolution, said image signal processingdevice comprising:means for decoding the encoded data recorded on saidrecording medium to obtain restored quantized orthogonal transformationcoefficients of said first image data and said supplementary data; meansfor inverse-quantizing said restored quantized orthogonal transformationcoefficients to obtain restored orthogonal transformation coefficientsof said first image data and said supplementary data; means for applyingan inverse-orthogonal transformation to said restored orthogonaltransformation coefficients to restore said first image data and saidsupplementary data; means for transforming said restored first imagedata to restored expanded image data; means for reproducing a thirdimage data based on said restored expanded image data and said restoredsupplementary data; and means for outputting said restored third imagedata to a second display device showning an image at a second resolutionhigher than said first resolution.
 16. A device according to claim 15,further comprising:means for determining whether said supplementary datais recorded to said recording medium, said reproducing means reproducingsaid third image data according to recognition information.